Mixtures of probability experts for audio retrieval and indexing

This paper describes a system for connecting non-speech sounds and words using linked multi-dimensional vector spaces. An approach based on mixture of experts learns the mapping between one space and the other. This paper describes the conversion of audio and semantic data into their respective vector spaces. Two different mixture-of-probability-expert models are trained to learn the association between acoustic queries and the corresponding semantic explanation, and visa versa. Test results are presented based on commercial sound effects CDs. 1. THE APPROACH This paper describes a method of connecting sounds to words, and words to sounds. Given a description of a sound, the system finds the audio signals that best fit the words. Thus, a user might make a request with the description " the sound of a galloping horse, " and the system responds by presenting recordings of a horse running on different surfaces, and possibly of musical pieces that sound like a horse galloping. Conversely, given a sound recording, the system describes the sound or the environment in which the recording was made. Thus, given a recording made outdoors, the system says confidently that the recording was made at a horse farm where several dogs reside. A system that has these functions, called MPESAR (mixtures of probability experts for semantic–audio retrieval), learns the connections between a semantic space and an acoustic space. Semantic space maps words into a high-dimensional probabilis-tic space. Acoustic space describes sounds by a multidimen-sional vector. In general, the connection between these two spaces will be many to many. Horse sounds, for example, might include footsteps and neighs. Figure 1 shows one half of MPESAR: how to retrieve sounds from words. Annotations that describe sounds are clustered and represented with multinomial models. The sound files, or acoustic documents, that correspond to each node in the semantic space are modeled with Gaussian mixture models (GMMs). Given a semantic request, MPESAR identifies the portion of the semantic space that best fits the request, and then measures the likelihood that each sound in the database fits the GMM linked to this portion of the semantic space. The most likely sounds are returned to satisfy the user's semantic request. Figure 2 shows the other half of MPESAR: how to generate words to describe a sound. MPESAR analyzes the collection of sounds and builds models for arbitrary sounds. This approach gives us a multi-dimensional representation of any sound, and a distance …